1. Field of Invention
Aspects of the present invention are directed to color image processing, and more particularly to modified Retinex image processing.
2. Discussion of Related Art
The quality provided by digital technologies has created a huge demand for digital products in general. Part of this digital revolution is the increased popularity of digital images. At the heart of this digital image revolution are image processing systems. These systems process the captured digital image to enhance the clarity and details of the image using sophisticated image processing algorithms. The use of these algorithms result in images that are substantially more accurate and detailed than previously achieved using older analog methods.
However, when compared to the direct observation of scenes, color images in general have two major limitations due to scene lighting conditions. First, the images captured and displayed by photographic and electronic cameras suffer from a comparative loss of detail and color in shadowed and highlighted zones. This is known as the dynamic range problem. Second, the images are subject to color distortions when the spectral distribution of the illuminant changes. This is known as the color constancy problem. A commonly encountered instance of the color constancy problem is the spectral difference of the same scene illuminated by daylight and artificial light, for example, by tungsten light, which will result in different shading.
The human visual system is typically able to adapt to lighting variations across scenes, visualizing details in regions over a very wide range of illuminations. Most image recording systems, however, fail to be able to tolerate such a wide range in lighting variation, and therefore cannot accurately capture the full dynamic range of the scene. As a result, images produced by these image recording systems are often of poor quality, compared to images produced by human perception. Another task that is often poorly performed by the image recording systems is that of color constancy. Humans perceive color in a way that is fairly independent of scene illumination, whereas the image recording systems are strongly influenced by spectral shifts.
The above problems can be stated mathematically by describing a relationship between an acquired image S, a reflectance of objects of the image R, and an illumination L in a pixel-wise multiplication, or:S=R×L  Equation (1)
This expression means that at each point in the image S, the color value is the multiplication of the reflectance value by the illumination value. Given an image S, the problem to be solved is removal of the effects of illumination and recovery of the reflectance image. That is, given S, find both R and L. However, calculation of both R and L is typically not possible. The challenge then, is to generate a methodology that can estimate R and L.
In the field of image processing, a center/surround Retinex process has been developed in which functions of the retina and cortex are substantially modeled. The term Retinex is coined from mixing the words retina and cortex, since both play an important role in the vision process.
Retinex processing deals with correcting image information for illumination in the images. The visual sensation is modeled and surrounding information with respect to a center pixel is taken into consideration. Collectively this is referred to as “Retinex Processing.”. The goal of the Retinex theory is to decompose a given image S into the reflectance image R, and the illumination image L, such that at each point (x,y) in the image domain, S(x,y) equals the product of R(x,y) and L(x,y). The benefits of such decomposition include the possibility of removing illumination effects of back/front lighting, enhancing photographs and other image capture methods that include spatially varying illumination, such as images that contain indoor and outdoor zones, and correcting colors in images by removing illumination-induced color shifts.
As noted above, recovering the illumination L from a given image S is known to be a mathematically ill-posed problem, and known algorithms vary in the manner and effectiveness of overcoming this limitation. The Retinex approach provides the framework for one such method. The Retinex methodology was motivated by Edward Land's research of the human visual system, which is described in R. H. Land, “The Retinex Theory of Color Vision,” Sci. Amer., Vol. 237, pp. 108-128 (1977).
The first Retinex algorithms were of the random walk type. Subsequent Retinex algorithms used homomorphic filtering. Homomorphic filtering is a generalized technique for image enhancement. It simultaneously normalizes the brightness across an image and increases contrast. Since illumination and reflectance are not separable, but their approximate locations in the frequency domain may be located, homomorphic filtering separates the components by processing the natural logarithm of the image in the frequency domain. To make the illumination of an image more even, the high-frequency components are increased and low-frequency components are decreased because the high-frequency components are assumed to be the reflectance in the scene (the amount of light reflected off the object in the scene), whereas the low-frequency components are assumed the illumination in the scene. Filtering is used to separate and repress low frequencies, which is assumed to be the illumination component, and high pass filtering is used to amplify high frequencies, which is assumed the surface reflection component.
The application of Land's human vision theories to image processing has been attempted in the prior art. Central to the prior art Retinex methods is the Retinex-type algorithm, discussed above with respect to Equation (1), namely that a perceived image S is the multiplication between the illumination L shed on visible surfaces and the respective reflectance R of the surfaces. An underlying assumption behind Retinex algorithms is that the illumination L is an artifact. The illumination L is estimated and either removed completely, or partially. Estimating L from S is the main algorithmic and computational problem in Retinex algorithms.
Prior art Retinex-type algorithms are characterized by a two-module structure as shown in FIG. 1. An illumination estimation module 10 computes a smooth version L (i.e., the local statistics) of an input image S. Usually the smooth version L is a either a linear or a non-linear low-pass filter of the input image S. A manipulation module 20 then manipulates pixels in image S according to correspondingly determined values of L. In the case of Retinex-type algorithms, the L is usually a local average or local envelope (maximum) of S. Sometimes the L may be a robust local average or robust local envelope of S, whereby robust means that pixels participating in determining the local average or envelope are on the same side of perceptually significant image edges as the center pixel.
In FIG. 1, for convenience, the input image S is shown as the input to the Retinex algorithm. However, as is known to those of ordinary skill in the art, Retinex-type algorithms typically operate in the Log domain. As is also know to those of ordinary skill in the art, the illumination L is often referred to as an “envelope.” The envelope can be smooth or piece-wise smooth.
Prior art Retinex algorithms also typically use linear space invariant low pass filters or partial differential equations for the illumination estimation module 10. Variants include slowly varying envelopes, i.e., local envelopes instead of local averages, and robust low passes resulting in piece-wise smooth averages or envelopes, which might change abruptly whenever the input changes abruptly.
In the illumination manipulation module 20 module, the illumination L might be subtracted in part, for example, subtract half of the illumination L from the input image S. Alternative manipulation methods may reduce more of the input image S values as corresponding illumination L values increase.
Prior art Retinex algorithms may be applied to monochrome or color images. In the color image case, the Retinex algorithm may be applied to all planes, e.g. the three color planes that comprise the input image signal.
In some prior art Retinex algorithms both the illumination estimation and illumination manipulation modules 10 and 20 are performed in an iterative filtering and subtraction scheme. In other prior art Retinex algorithms, the modules 10 and 20 are interleaved.
In an improved prior art Retinex-type algorithm, as disclosed in published U.S. application Publication No. 2005/0074163A1 the illumination L′ is obtained from a sub-sampled version S′ of the input image S. Such an improved Retinex-type algorithm is shown in block diagram form in FIG. 2. In FIG. 2, the Retinex-type algorithm includes linear illumination estimation module 30 and the illumination manipulation module 20, as discussed with respect to FIG. 1. The image signal S is first input to a down sample module 32, where the image signal S is sub-sampled using techniques known to those of ordinary skill in the art to produce sub-sampled images S′. For example, the sub-sampling may involve averaging blocks of four pixels in the image S. The linear illumination estimation module 30 then generates an estimate of the illumination L′ based on the sub-sampled image S′. The estimate of the illumination L′ is then provided to an up-sample module 34, and an estimation of the illumination L″ of the entire image S is produced, using interpolation and similar techniques known to those of ordinary skill in the art. The estimate of the illumination of the entire image L″ is then operated on as discussed above with respect to FIG. 1 to provide an estimate of the reflectance R of the entire image.
The sub-sampling and subsequent interpolation as discussed with respect to FIG. 2. are intended to speed up the computationally intensive Retinex process. However, in the Retinex-type algorithm illustrated in FIG. 2, the interpolation is performed on a set of smooth, low resolution intermediate images (i.e., the images S′), using the high resolution input image S to select corresponding output pixels from the resulting set of high resolution images L″. Furthermore, the computationally intensive operation of the Retinex-type algorithm is repeated several times, once for each of the intermediate images S′, e.g. for the three color planes that comprise the input image signal.